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Summary: The publication in 2003 of the Ranking of Universities by Jiao Tong University of 
Shanghai has revolutionized not only academic studies on Higher Education, but has also had 
an important impact on the national policies and the individual strategies of the sector. The work 
gathers the main characteristics of this and other global university rankings, paying special 
attention to their potential benefits and limitations. The Web Ranking is analyzed in depth, 
presenting the model on which its compound indicator is based and analyzing its different 
variables. 
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Introduction 

Before 2003, studies on Higher Education had a limited scope, with study populations 
that were either small or geographically or culturally biased. In most cases, it was an analysis of 
the situation in North America or Western Europe, with special attention paid to the model defined 
by North American universities intensely focused on research. The universities from the rest of 
the world were the subject matter merely of specific studies and those from developing countries 
rarely deserved any attention. 

On the other hand, the classifications or rankings available were national in nature, 
frequently made by large-circulation newspapers or magazines, which purpose was to inform or 
guide future students when choosing a university on where to start or continue their studies. The 
criteria used merely described academic offerings and focused on the costs of education, the 
quality of life on campus or in the city, security and the offering of services of universities 
(Eccles, 2002). These criteria based, generally, on subjective perceptions, vague sources or 
very biased data. 

Basically, there were no world rankings before the publication of what was known as the 
Shanghai Ranking (www.shanghairanking.com), the "Academic Ranking of World Universities" 
(ARWU), made by scientists from Jiao-Tong University of Shanghai, in the People's Republic of 
China (Liu & Cheng, 2005). In 2003, the first edition of this annual ranking was published. Its 
main novelties are: 
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• A list of the top 500 universities in the world, with more than 40 countries represented. 

• A compound index combining variables related to the excellence of the institution, mainly 
regarding research results. 

• The ranking is made by researchers from a public university based on openly available 
data and sources. 

• Open publication on the Web, with a very simple consultation interface. It uses English, 
the lingua franca of the scientific community, as the language of the texts. 

The Shanghai Ranking impact was immediate and profound. The reactions were diverse, 
but after some criticism (van Raan, 2005), different countries started to take political measures 
to adapt their national university systems in order to improve their results in the mentioned 
Ranking. Individual institutions also took similar measures and it can be said that the ranking has 
attained a level of transcendence and relevance that has a truly global reach (Marginson & van 
der Wende, 2007). 

That impact seems to fly in the face of the rigorous and verified criticism in scientific 
articles that the Ranking received and continues to receive (Billaut, Bouyssou, & Vinke, 2009). 
Actually, in order to maintain the methodological stability and comparability among editions, the 
Shanghai Ranking has not corrected practically any of the problems that were found in it. 

Among the most prominent: 

• The lack of specific indicators for technologies, social sciences and humanities, which 
forces specific treatment of certain very prestigious institutions (for example, the London 
School of Economics). 

• The bibliometric biases in favor of biomedical institutions, some of which do not have a 
clear university status. 

• The use of very old series for certain variables. Nobel Prizes were considered from their 
inception in 1900, although with decreasing importance according to the decade. 

• The use of the alma mater of the winners (Nobel or Field medals) as a measure of 
teaching quality. 

• The lack of very relevant awards, besides the mentioned Nobel and Field, especially in 
disciplines not included in them. 

• The exclusion of the universities of Berlin due to an alleged difficulty in assigning values 
from before World War II. 

• The acceptance of secondary institutions among the highly quoted scientists’ affiliations 
(visiting professors in Saudi universities). 

• The combination of absolute and relative indices. 

Global Ranking of Universities 

After the publication of the Shanghai Ranking, in light of its impact and taking advantage 
of certain omissions, several other rankings have been published, with the goal of having global 
coverage. They can be divided into two large groups, excluding from the list those that are 
anonymous, have no clear methodological basis or show clear biases in their results: 
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Rankings based on opinion surveys. They are two very similar rankings, derived from a 
worked performed by the British company QS for the magazine Times Higher Education from 
2004 and that as of 2010 generated two different products produced by each of these 
organizations. The core of the results is obtained from opinion surveys and complemented with 
bibliometric data. QS uses the Scopus/Elsevier database, while THE is based on the information 
provided by Thomson/Reuter's Web of Knowledge (WoK). 

Rankings based on Bibliometrics/Cybermetrics. There are four rankings that use, almost 
exclusively, quantitative data derived from their research results (scientific articles and 
bibliographic citations) or their Web presence (Web pages, links and mentions). The HEEACT 
ranking from Taiwan and the CWTS (University of Leiden, Holland) rely on the WoK data, while 
the ranking of the Scimago group (Spain) uses its competitor, Scopus. The Web Ranking or 
Webometrics share methodology but not sources and it uses, apart from data from search 
engines, information extracted from Google Scholar, a bibliographic database of citations similar 
to Scopus and Wok. 

Just as with the Shanghai Ranking, these classifications have also been the object of 
criticism, although after more than 8 years of experience, most of them have managed to become 
part of political agendas and higher education strategies at a global level. As a matter of fact, 
today it is not uncommon for them to be used practically to validate degrees, grant student loans 
or even at parliamentary debates. 

However, not all of them are the same and it is necessary to highlight both their 
strengths and weaknesses: 

• QS Ranking (www.topuniversities.com). This is one of the most commonly used rankings. 
It provides information on more than 700 universities and offers rankings by region (Asia, 
Latin America) and by discipline. Strongly biased in favor of universities in countries where 
English is the main or principal language, it also over represents Asian countries. This 
was due to methodological problems with the surveys used and it appears to have been 
solved in the latest editions. As in other rankings, the universities occupying the last 
positions, beyond position 400, are not trustworthy. 

• THE Ranking (www.timeshighereducation.co.uk/world-university-rankings). Presented as 
an evolution and improvement of the former, it includes several extra variables, 
whose relative contribution is quite subjective. The first editions have suffered from 
significant problems in their bibliometric calculations and from the lack of transparency 
about the universities actually evaluated. The latest version is still inadequate. 

• Leiden Ranking (socialsciences.leiden.edu/cwts/products-services/leiden-ranking-2010- 
cwts.html). This classification is produced by what is possibly the group that is the global 
leader in bibliometric research. The quality of the data is very good. However, 
unfortunately, five different rankings are presented that use exactly the same source 
information. The authors do not explicitly favor any of their own proposals and are also 
developing the main indicator, called the "Crown indicator", into a new one which is in 
fact quite different. 
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• Taiwan Raking (ranking.heeact.edu.tw). Another exclusively bibliometric ranking, 
developed from a combination of variables that are strongly correlated and that does not 
seem to respond to a mixed model. 

• Scimago Ranking (www.scimagoir.com). The only one that includes both universities 
and research centers, with a total of 3,000 institutions analyzed. Although it provides 
values for six different variables, some of which are only available here, the order of 
presentation is derived solely from the number of scientific publications according to the 
Scopus database. This lack of a true "ranking", as well as the biases in Scopus' 
coverage, render the comparability of its results with other rankings difficult. 

• Webometrics Ranking (www.webometrics.info). This ranking will be discussed in detail 
below, but we can indicate, among its advantages, its broad coverage, both in number of 
universities (20,000) and in academic missions (teaching, research, commitment to the 
community, transfer of technology, internationalization). However, since it is a web- 
based ranking, it is highly dependent on the application of proper politics by the 
universities and discrepancies due to poor practices in matters of Web presence are not 
uncommon. 


Results Comparison 

It is difficult to perform an in-depth analysis of all these rankings without using 
sophisticated statistical techniques or political-economic considerations (Aguillo et al., 2010; Dill 
& Soo, 2005). However, by choosing a wide comparison level, in this case the countries 
represented among the 500 best institutions according to the different classifications, it is 
possible to illustrate the differences among them and at the same time offer a candidate for a 
global vision of the worldwide university scenario. 

Table 1 shows the data corresponding to the last available editions (2011, except for the 
Leiden Ranking, for which the 2010 data were used) with the following specifications: the 
selected version of the Leiden Ranking (CWTS) has been the so-called "brute force" (total 
publications multiplied by the crown indicator). The Scimago data have been combined in a 
proportion of 1:1 (50%:50%) between publications and citations, after being standardized. For 
the QS Ranking, only the 498 universities appearing on their Website were used. The THE 
ranking has not been included in the analysis. For comparability purposes, the three bibliometric 
rankings are shown consecutively. 

At the country level, we can study the relative coverage. A total of 59 countries (including 
Hong King and Taiwan) appear on the combined list. QS is the ranking with the widest coverage 
(50), while Taiwan (HEEACT), which only includes 39, is the ranking with the least diversity. 
Thirteen countries appear in only one of the rankings, mostly presented by QS (8), while 
Webometrics has two and Leiden, Scimago and Shanghai one each. From these data, it can be 
deduced that QS is the ranking that deviates the most from the average behavior, although that 
possibly has a fundamental influence on the last positions of the countries included. 
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Table 1. Number of universities per country among the top 500 classified according to different world 
rankings 


Country 

ARWU 

11 

WRJUL 

11 

HEEACT 

11 

CWTS 

10 

SCIM 

11 

QS 

2011 

MEDIAN 

AVERAGE 

MAX. 

USA 

151 

172 

158 

135 

142 

103 

150 

152 

172 

Germany 

39 

47 

46 

46 

41 

41 

46 

45 

47 

United 

Kingdom 

37 

37 

36 

38 

38 

52 

38 

37 

38 

Italy 

22 

17 

29 

32 

27 

15 

28 

26 

32 

Japan 

23 

12 

24 

24 

25 

24 

24 

21 

25 

Canada 

22 

24 

21 

19 

22 

20 

22 

22 

24 

France 

21 

9 

22 

23 

14 

21 

18 

17 

23 

China 

23 

8 

18 

13 

32 

16 

16 

18 

32 

Spain 

11 

26 

13 

20 

14 

13 

17 

18 

26 

Australia 

19 

12 

13 

9 

13 

23 

13 

12 

13 

Holland 

13 

9 

12 

12 

12 

13 

12 

11 

12 

South 

Korea 

11 

3 

10 

8 

14 

12 

9 

9 

14 

Sweden 

11 

9 

11 

10 

10 

8 

10 

10 

11 

Belgium 

7 

7 

7 

7 

7 

7 

7 

7 

7 

Switzerland 

7 

7 

8 

7 

7 

8 

7 

7 

8 

Taiwan 

7 

11 

6 

6 

7 

10 

7 

8 

11 

Finland 

5 

6 

6 

7 

6 

7 

6 

6 

7 

Austria 

7 

5 

6 

6 

5 

5 

6 

6 

6 

Brazil 

7 

12 

5 

5 

7 

4 

6 

7 

12 

Portugal 

2 

6 

4 

6 

5 

3 

6 

5 

6 

Hong Kong 

5 

6 

5 

5 

5 

6 

5 

5 

6 

Israel 

7 

4 

5 

7 

5 

4 

5 

5 

7 

Norway 

4 

4 

4 

4 

3 

4 

4 

4 

4 

Denmark 

4 

5 

4 

4 

4 

5 

4 

4 

5 

Greece 

2 

3 

4 

6 

4 

3 

4 

4 

6 

South 

Africa 

3 

1 

2 

2 

3 

3 

2 

2 

3 

Ireland 

3 

4 

3 

3 

3 

7 

3 

3 

4 

Singapore 

2 

2 

2 

2 

2 

2 

2 

2 

2 

Chile 

2 

1 

2 

1 

2 

3 

2 

2 

2 

Thailand 

0 

4 

1 

0 

2 

2 

2 

2 

4 

Humgary 

2 

3 

2 

4 

1 

1 

3 

3 

4 

Argentina 

1 

2 

1 

2 

1 

4 

2 

2 

2 

Russia 

2 

1 

1 

2 

1 

7 

1 

1 

2 

India 

1 

0 

2 

1 

2 

7 

2 

1 

2 

Poland 

2 

3 

2 

6 

2 

2 

3 

3 

6 

New 

Zealand 

5 

2 

2 

2 

2 

6 

2 

2 

2 

Turkey 

1 

1 

0 

9 

3 

2 

2 

3 

9 
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Slovenia 

1 

1 

1 

1 

1 

0 

1 

1 

1 

Croatia 

1 

0 

0 

1 

1 

0 

1 

1 

1 

Mexico 

1 

1 

1 

1 

2 

2 

1 

1 

2 

Saudi 

Arabia 

2 

2 

0 

0 

0 

3 

0 

1 

2 

Czech 

Republic 

1 

4 

1 

2 

1 

1 

2 

2 

4 

Malaysia 

1 

3 

0 

0 

0 

5 

0 

1 

3 

Estonia 

0 

1 

0 

0 

0 

0 

0 

0 

1 

Iceland 

0 

1 

0 

0 

0 

0 

0 

0 

1 

Slovakia 

0 

1 

0 

1 

0 

0 

1 

1 

1 

Serbia 

1 

0 

0 

0 

1 

0 

0 

0 

1 

Iran 

0 

0 

0 

0 

1 

0 

0 

0 

1 

Uruguay 

0 

0 

0 

0 

0 

1 

0 

0 

0 

Lebanon 

0 

0 

0 

0 

0 

1 

0 

0 

0 

Oman 

0 

0 

0 

0 

0 

1 

0 

0 

0 

Pakistan 

0 

0 

0 

0 

0 

1 

0 

0 

0 

United 

Arab 

Emirates 

0 

0 

0 

0 

0 

1 

0 

0 

0 

Estonia 

0 

0 

0 

1 

0 

0 

0 

0 

1 

Egypt 

1 

0 

0 

0 

0 

0 

0 

0 

0 

Kazakhstan 

0 

0 

0 

0 

0 

2 

0 

0 

0 

Philippines 

0 

0 

0 

0 

0 

2 

0 

0 

0 

Colombia 

0 

1 

0 

0 

0 

2 

0 

0 

1 

Indonesia 

0 

0 

0 

0 

0 

3 

0 

0 

0 


Note: ARWU: Shanghai; WR: Webometrics; HEEACT: Taiwan; CWTS: Leiden; SCIM: Scimago; QS: QS. 

In order to study the behavior of QS in greater depth, it is necessary to rely on the 
individual countries, especially those occupying the first positions. The ranking that differs the 
most from the median for 24 countries where this value is greater than two has been identified. 
QS is the ranking that deviates the most in 11 cases (including the U.S.A., United Kingdom, 
Australia, Ireland, India or New Zealand, all countries where English is the official or principal 
language), followed by Webometrics, whose 7 countries include, precisely, examples where 
English is not the main language (Japan, France, Spain, South Korea or Brazil). Bibliometric 
rankings differ, as it can be expected, quite less and the overrepresentation of China in Scimago 
or Turkey in Leiden are noteworthy, as is the poor presence of Germany in the Shanghai 
Rankings. 

Some explanations are possible. The bias of the QS towards universities in countries 
with English as the official or principal language is perhaps the consequence of the uneven 
coverage of the survey. As for Webometrics, the discrimination of English is penalized by global 
audiences of the Web. Scopus, the database used by Scimago, may have a wider coverage of 
the Chinese periodical publications, while Leiden has been able to perform a better job identifying 
Turkish contributions. Finally, the unjustifiable exclusion of the universities of Berlin in Shanghai 
reflects on the cited result. 


18 


I. Aguillo 



















































High. Learn. Res. Commun. 


Vol. 2, Num. 1 | March 2012 


Cybermetrics and the Web Ranking 

The Web Ranking of Universities (Aguillo, Ortega & Fernandez, 2008) has been published 
since 2004, and is directly inspired by the compound indicator model of the Shanghai Ranking, 
but using cybernetic data taken from the university websites. Cybermetrics or Webometrics is an 
emerging discipline developed in the mid-90s that aims to describe in a quantitative manner the 
scientific communication processes and the structure of academic and research units from the 
current information or the information exchanged via the Internet (Aguillo et al., 2006). The 
Web is explored using web crawlers. The data can be obtained directly or through commercial 
search engines that, today, are the most powerful and efficient tools to describe global scenarios. 

The Web Ranking uses one of the most complete and updated university catalogs on 
the Web, as it studies more than 20,000 institutions of higher education from around the world. 
Figure 1 shows the Ranking's relative coverage by region. 



Figure 1. World and regional coverage of the University Web Ranking, with the indication of the 
institutions occupying the first thousand positions (July 2011 data). 

The Web Ranking uses a compound indicator model that keeps a 1:1 proportion between 
measures of activity or Web presence and those of visibility or Web impact (Aguillo, 2009). This 
means that each one of them contributes 50% to the weight of the indicator total. For practical 
purposes, four variables, defined as follows, are used. 

• Size. Number of Web pages of a university domain, obtained from search engines. 

• Rich files. Number of rich format documents (pdf, doc, ppt, ps) related to the activity of 
teaching, research or administration. Also obtained through search engines. 

• Scholar. Number of research articles or papers included in the Google Scholar and 
Scimago databases. 
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• Visibility. Number of external links and mentions to the university Web domain. 

The first three values are combined almost in equal proportion (20%, 15% and 15% plus 
a round up to improve legibility) to complete the 50% corresponding to the activity, while the 
other 50% corresponds to the virtual referendum that the visibility measure implies. 

Among the advantages of this method, we highlight the possibility of including all the 
missions of the university, provided that they are reflected on the Web. However, the simplicity 
of the rankings system makes it difficult to know with precision the contribution or relative 
importance of each of the missions; hence, an integrated perspective, in which additional works 
are necessary to identify strengths and weaknesses, is always offered. 

Among the main inconveniences of the Web Ranking, we point out their great 
vulnerability to poor practices in the design of Web presence strategies. Several hundred 
universities have more than one central Web domain, which divides the impact of their presence, 
apart from being a confusing and unpractical method. In other cases, the domains are changed 
with no contingency plan to keep the visibility of the original and, frequently, with no apparent 
or logical reason for those changes. Some universities make their scientific archives available to 
other organizations for study under a domain other than that of the university. Design problems, 
poorly organized content, a highly centralized structure, and lack of international information (in 
English) are some of the other causes for having lower positions in the Ranking. 

The results obtained closely correlate with those obtained through other rankings, 
especially if the mentioned discrepancies are excluded. However, there is a characteristic 
pattern that is only revealed by the Web Ranking: the existence of an academic digital gap 
between North American (United States and Canada) universities and their European 
counterparts. As shown in Table 2, there are many North American institutions among the top 
100 or 200, even twice or three times as much as from the rest of the world. By considering the 
top 500, those inequalities disappear, which indicates that it is the leading institutions that, apart 
from a strong commitment with Web publishing, create a network that includes universities 
highly active in research. 


Conclusions 

In summary, we may recommend that global rankings should not be ignored, since they 
fulfill an important mission and are an excellent tool in the design of strategies and university 
policies. A global critique is absurd, but the Rankings whose methodologies are not very 
academic, those with unjustified biases and those with commercial interests must be identified. 

The Web is the most important scientific communication tool and, therefore, Web 
presence is an indicator that reflects the university's overall performance, not only its online 
activities. The Web significantly increases the audience, the visibility and the impact of the 
activities and achievements of a university's professors, researchers and students. 
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Table 2. Academic digital gap among regions and countries through their presence in the Web Ranking of 
Universities (July 2011 issue) 

Region/Country 

Top 100 

Top 200 

Top 500 

Top 1000 

NORTH AMERICA 

73 

111 

196 

394 

USA 

67 

95 

172 

356 

Canada 

6 

16 

24 

38 

EUROPA 

16 

58 

221 

413 

United Kingdom 

7 

10 

37 

67 

Germany 

2 

12 

47 

66 

Holland 

2 

3 

9 

13 

Italy 

1 

3 

17 

37 

Switzerland 

1 

3 

7 

10 

ASIA 

7 

19 

49 

108 

Taiwan 

3 

6 

11 

21 

Japan 

2 

6 

12 

33 

China/Hong 

1 

5 

14 

19 

Kong 





OCEANIA 

2 

6 

14 

35 

Australia 

2 

6 

12 

28 

LATIN AMERICA 

2 

5 

17 

40 

Brazil 

1 

4 

12 

18 

Mexico 

1 

1 

1 

5 

ARAB WORLD 

1 

1 

1 

5 

AFRICA 



1 

5 


The Web is the future and current channel for distance education, the most economic 
and attractive window for recruiting the best international students and professors and the 
meeting point among the university, society, its economic players and political leaders. 

The Web Raking is the one with the widest coverage, since it classifies both universities 
at a world level and those relevant for the development of emerging countries. The discrepancies 
observed show us serious problems of governance, lack of long-term strategies, inadequate or 
obsolete policies and poor Web practices. 

The digital gap between the universities of the U.S.A. and those of the rest of the world 
is troubling, since it implies a scientific and cultural colonialism that will have profound 
consequences in the future. 
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